EDA Final Report

Introduction

In this section, explain why you chose this topic, and the questions you are interested in studying. Include a brief description of how you found the data, and clear instructions on where the reader can find the data.

International travel has become so easy that anyone can just grab his/her bags and documents and starts a new journey. As one of the most tourists importing and exporting countries, the USA has always played a significant role in world’s tourism industry. Therefore, we want to study people who traveled to and from the USA by investigating the Travel and Tourism Statistical data provided by The National Travel & Tourism Office (NTTO). To be more precise, we want to answer following questions:

• What are the most popular international travel destinations for Americans?

• What are the main regions of people who travel to the USA?

• What are the purposes and spending of travels to and from the USA?

• Does the economy matter for the answers to above questions?

Data is collected from two resources,

• National Travel & Tourism Office (NTTO):
  • Inbound Information
  • Outbound Information
• World Bank:

http://tinet.ita.doc.gov/outreachpages/inbound.general_information.inbound_overview.asp

http://tinet.ita.doc.gov/view/m-2016-I-001/index.asp

http://travel.trade.gov/research/monthly/arrivals/index.asp

NTTO monthly spending NTTO yearly spending NTTO monthly inbound NTTO monthly outbound Word Bank GDP

From NTTO, … NTTO do not provide raw data

Although NTTO provides plenty of information we need, it is not easy to collect and combine them together. There is much unavoidable manual work to do in the collecting process because of the inconsistent formatting (data formats may vary from year to year), file types (some are pdfs) and location of these files. For example, in order to get monthly arrival information per region, we have to download each year’s excel files, and then clean, select and combine them into a final master table that is easier to manipulate with in R.

In the end, we collect several master tables from NTTO: yearly spending per region with three categories, total monthly spending with three categories, monthly inbound visitation per country and monthly outbound visitation per country. We may use other tables as complementary tables.

From World Bank,… On this website, we are trying to get GDP per country per year, as a measurement of economy.

GDP data from World Bank is easier to clean. Further, we will load these tables to R and use dplyr and tidyr to fit the data for our analysis.

Team

List team members and a description of how each contributed to the project. (If you’re working alone, briefly describe the stages of the project.)

• Lucy Lu: finding and organizing the data, compiling visualization and stories, creating the interactive app, focusing on the import part.

• Hanlin Wu: cleaning the data, testing various visualizations, creating the interactive app, focusing on the export part.

Analysis of Data Quality

Provide a detailed, well-organized description of data quality, including textual description, graphs, and code.

library(tidyverse)
library(viridis)
library(ggvis)
library(extracat)
library(plotly)
library(zoo)

raw_ntto_spend_y <- tbl_df(read.csv("clean_data/clean_yearly_spending_region.csv", header = TRUE))
# raw_ntto_spend_m <- tbl_df(read.csv("clean_data/Monthly_Exports_Imports_Balance.csv", header = TRUE))
raw_ntto_inbound_m <- tbl_df(read.csv("clean_data/clean_monthly_visitation_inbound_country.csv", header = TRUE))
raw_ntto_outbound_m <- tbl_df(read.csv("clean_data/clean_monthly_us_to_international.csv", header = TRUE))
raw_wb_gdp <- tbl_df(read.csv("clean_data/API_NY.GDP.MKTP.CD_DS2_en_csv_v2.csv", header = TRUE, skip=4))

Data we have are: monthly spending from NTTO, yearly spending from NTTO, monthly inbound from NTTO, monthly outbound from NTTO and GDP from Word Bank. In order to analyze data quality of our data, we need first to take a look at them.

Output of str has been truncated to get a more compact look in report

• Yearly Spending from NTTO

str(raw_ntto_spend_y, list.len=5)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   48 obs. of  13 variables:
 $ region  : Factor w/ 8 levels "Africa","Asia-Pacific",..: 1 1 1 1 1 1 2 2 2 2 ...
 $ type    : Factor w/ 2 levels "Payments (imports)",..: 2 2 2 1 1 1 2 2 2 1 ...
 $ category: Factor w/ 3 levels "Education","Medical/Short-Term Workers",..: 1 2 3 1 2 3 1 2 3 1 ...
 $ X2006   : int  930 208 1390 137 29 1740 8566 836 23160 626 ...
 $ X2007   : int  961 183 1594 161 30 2435 9455 810 24756 667 ...
  [list output truncated]

• Monthly Spending from NTTO

# str(raw_ntto_spend_m, list.len=4)

• Monthly Inbound from NTTO

str(raw_ntto_inbound_m, list.len=5)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   132 obs. of  32 variables:
 $ year                       : int  2015 2015 2015 2015 2015 2015 2015 2015 2015 2015 ...
 $ MONTH                      : Factor w/ 12 levels "APRIL","AUGUST",..: 5 4 8 1 9 7 6 2 12 11 ...
 $ TOTAL.ARRIVALS             : Factor w/ 132 levels "2,592,882","2,624,018",..: 91 70 109 119 124 110 130 131 120 121 ...
 $ CANADA                     : Factor w/ 132 levels "1,007,315","1,029,277",..: 63 39 109 96 89 50 112 120 71 44 ...
 $ MEXICO                     : Factor w/ 132 levels "1,004,126","1,007,303",..: 44 18 27 50 52 38 53 57 35 55 ...
  [list output truncated]

• Monthly Outbound from NTTO

str(raw_ntto_outbound_m, list.len=4)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   102 obs. of  14 variables:
 $ Year   : int  2015 2015 2015 2015 2015 2015 2015 2015 2015 2015 ...
 $ Regions: Factor w/ 17 levels "Africa","Asia",..: 8 6 2 16 7 15 13 1 17 12 ...
 $ Jan    : Factor w/ 102 levels "1,707,635","1,716,026",..: 96 91 59 16 47 89 20 49 31 37 ...
 $ Feb    : Factor w/ 102 levels "1,514,915","1,580,058",..: 79 94 59 16 50 98 17 39 30 35 ...
  [list output truncated]

• GDP from Word Bank

str(raw_wb_gdp, list.len=6)

Classes ‘tbl_df’, ‘tbl’ and 'data.frame':   264 obs. of  62 variables:
 $ Country.Name  : Factor w/ 264 levels "Afghanistan",..: 11 1 6 2 5 8 250 9 10 4 ...
 $ Country.Code  : Factor w/ 264 levels "ABW","AFG","AGO",..: 1 2 3 4 5 6 7 8 9 10 ...
 $ Indicator.Name: Factor w/ 1 level "GDP (current US$)": 1 1 1 1 1 1 1 1 1 1 ...
 $ Indicator.Code: Factor w/ 1 level "NY.GDP.MKTP.CD": 1 1 1 1 1 1 1 1 1 1 ...
 $ X1960         : num  NA 5.38e+08 NA NA NA ...
 $ X1961         : num  NA 5.49e+08 NA NA NA ...
  [list output truncated]

The internal structure of these data indicates several data quality issues:

• Not tidy:
  • It is better in our case to have a columns named Year instead of each year has its own column.
• Messy column names:
  • Some columns like X1960 is weird and we should remove the extra “X”.
• Wrong types of values:
  • Some numerical value have been recognized as string (e.g. “1,707,635”, “$1,372”)
• Inconsistency of categorical variable values and column names:
  • category in Yearly Spending and X.1 (The content of this column represents spending category as well) in Monthly Spending have different number of factors and for the same category. For example, we suspisious that “Education related” and “Medical/Education/Workers Spending” should refer to the same category.
  • In some table month is represented as first three letters (e.g. Jan) and in other table is represented as full name (e.g. APRIL).
  • In consistency of upper/lower case of strings (e.g. Year and year)
• Missing values
  • In order to show missing value more clearly, we pre-processed data, the detail about how to clean data will be discussed later in Main Analysis part.
  • As what are shown in the graph below, we only have some missing values in GDP data for countries we are interested. But the missing values are either before 1990 nor after 2016, which would not influence our analysis.

country_str <- 'Argentina|Australia|Brazil|Canada|Chile|China|Colombia|France|Germany|India|Italy|Japan|Mexico|Netherlands|Russia|Saudi Arabia|South Korea|Singapore|Spain|Sweden|Switzerland|Taiwan|United Kingdom|United States' 
tidy_ntto_spend_y <- raw_ntto_spend_y %>%
   gather(x_year, spend, -region, -type, -category) %>% 
   separate(x_year, c("X", "Year"), sep="X") %>% 
   select(-X) %>% 
   select(Region=region, Type=type, Category=category, Year=Year,Spend=spend) %>%  
   mutate(Region=gsub("^\\s+|\\s+$", "", Region)) %>% 
   mutate(Missing = ifelse(is.na(Spend), "yes", "no"), Region=tolower(Region), Year=factor(Year), Region=factor(Region))
tidy_ntto_inbound_m <- raw_ntto_inbound_m %>%
   mutate(as.character(TOTAL.ARRIVALS)) %>% 
   gather(key, val, -year, -MONTH) %>%
   mutate(Arrival=as.numeric(gsub('[^0-9]+', '', val)), Month=match(tolower(substr(MONTH, 1, 3)), tolower(month.abb)), key=tolower(gsub('\\.', ' ', key)), Year=year) %>%
   filter(!grepl("total", key)) %>%
   unite(YearMonth, year, Month, sep="-") %>%
   select(Year, YearMonth, MixRegion=key, Inbound=Arrival) %>% 
   mutate(MixRegion=recode(MixRegion, 
                           "latin america  excl mexico "="latin america excl mexico",
                           "prc  excl hk "="china",
                           "roc  taiwan "="taiwan")) %>% 
   mutate(Missing = ifelse(is.na(Inbound), "yes", "no"), Year=factor(Year), Date=as.yearmon(YearMonth), MixRegion=factor(MixRegion)) %>% 
   arrange(MixRegion, Date)
tidy_ntto_outbound_m <- raw_ntto_outbound_m %>% 
   gather(Month, Outbound, -Year, -Regions) %>% 
   mutate(Outbound=as.numeric(gsub('[^0-9]+', '', Outbound)), Month=match(Month, month.abb), MixRegion=tolower(gsub('\\([a-zA-Z0-9]+\\)', '', Regions)), year=Year) %>% 
   mutate(MixRegion=gsub("^\\s+|\\s+$", "", MixRegion)) %>% 
   filter(!grepl("air", MixRegion), !MixRegion=='total overseas', !MixRegion=='north america', !MixRegion=='grand total') %>% 
   unite(YearMonth, year, Month, sep="-") %>% 
   mutate(Region=recode(MixRegion, 
                        "caribbean"="latin america excl mexico",
                        "central america"="latin america excl mexico",
                        "south america"="latin america excl mexico")) %>% 
   mutate(Missing=ifelse(is.na(Outbound), "yes", "no"), Year=factor(Year), Date=as.yearmon(YearMonth), MixRegion=factor(MixRegion), Region=factor(Region)) %>% 
   select(Region, MixRegion, Year, Date, Missing, Outbound) %>% 
   arrange(Region, MixRegion, Date)
tidy_wb_gdp <- raw_wb_gdp %>% 
   select(-Indicator.Name, -Indicator.Code, -X) %>% 
   gather(x_year, GDP, -Country.Name, -Country.Code) %>% 
   filter(grepl(country_str, Country.Name)) %>% 
   separate(x_year, c("X", "Year"), sep="X") %>% 
   select(-X) %>% 
   select(CountryName=Country.Name, CountryCode=Country.Code, Year=Year, GDP=GDP) %>% 
   # filter(year>1999) %>% 
   mutate(Missing = ifelse(is.na(GDP), "yes", "no"))

ggplot(tidy_ntto_spend_y, aes(x=Year, y=Region, fill = Missing)) +
   geom_tile(color = "white") + 
   scale_fill_viridis(discrete = TRUE) +
   ggtitle("Missing Value of NTTO Yearly Spending Data") +
   theme_linedraw()

ggplot(tidy_ntto_inbound_m, aes(x=Date, y=MixRegion, fill = Missing)) +
   geom_tile(color = "white") + 
   scale_fill_viridis(discrete = TRUE) +
   ggtitle("Missing Value of NTTO Inbound Data") +
   theme_linedraw() +
   theme(axis.text.x=element_text(angle=90))

ggplot(tidy_ntto_outbound_m, aes(x=Date, y=MixRegion, fill = Missing)) +
   geom_tile(color = "white") + 
   scale_fill_viridis(discrete = TRUE) +
   ggtitle("Missing Value of NTTO Outbound Data") +
   theme_linedraw() +
   theme(axis.text.x=element_text(angle=90))

ggplot(tidy_wb_gdp, aes(x=Year, y=CountryName,  fill = Missing)) +
   geom_tile(color = "white") + 
   scale_fill_viridis(discrete = TRUE) +
   ggtitle("Missing Value of All Interested Countries (1960-2016)") +
   theme_linedraw() +
   theme(axis.text.x=element_text(angle=90)) 

Executive Summary

Provide a short nontechnical summary of the most revealing findings of your analysis with no more than 3 static graphs or one interactive graph (or link), written for a nontechnical audience. The length should be approximately 2 pages (if we were using pages…) Do not show code, and take extra care to clean up your graphs, ensuring that best practices for presentation are followed.

• Note: the tips below are not intended to be a complete list of everything we’ve covered this semester on designing a successful graph. It’s meant to help you avoid some common problems.

• Title, axis labels, tick mark labels, and legends should be comprehensible (easy to understand) and legible (easy to read / decipher).

• Tick marks should not be labeled in scientific notation or with long strings of zeros, such as 3000000000. Instead, convert to smaller numbers and change the units: 3000000000 becomes “3” and the axis label “billions of views”.

• Units should be intuitive (Extreme example: an axis labeled in month/day/year format is intuitive, one labeled in seconds since January 1, 1970 is not.)

• The font size should be large enough to read clearly. The default in ggplot2 is generally too small. You can easily change it by passing the base font size to the theme, such as + theme_grey(16). (The default base font size is 11.)

• The order of items on the axes and legends is logical. (Alphabetical is often not logical.)

• Colors should be color vision deficiency friendly.

• If a legend is taking up too much space on the right, move it to the bottom.

• If categorical variable levels are long, set up the graph so the categorical variable is on the y-axis and the names are horizontal. A better option, if possible, is to shorten the names of the levels.

• Not all EDA graphs lend themselves to presentation, either because the graph form is hard to understand without practice or it’s not well labeled. The labeling problem can be solved by adding text in an image editor. The downside is that it is not reproducible. If you want to go this route, Paintbrush is a free and simple bitmap image editor for the Mac: https://paintbrush.sourceforge.io/ There are many other options.

• Err on the side of simplicity. Don’t, for example, overuse color when it’s not necessary. Ask yourself: does color make this graph any clearer? If it doesn’t, leave it out.

• Test your graphs on nontechnical friends and family and ask for feedback.

Main Analysis

Provide a detailed, well-organized description of your findings, including textual description, graphs, and code. Your focus should be on both the results and the process. Include, as reasonable and relevant, approaches that didn’t work, challenges, the data cleaning process, etc.

• The guidelines for the Executive Summary above do NOT apply to exploratory data analysis. Your main concern is designing graphs that reveal patterns and trends.

• As noted in Hmk #4, do not use circles, that is: bubbles, pie charts, or polar coordinates.

• Use stacked bar charts sparingly. Try grouped bar charts and faceting as alternatives, and only choose stacked bar charts if they truly do a better job than the alternatives for observing patterns.

Data Cleaning

Since the data is very messy, we put many effort on cleaning and extract useful infomation for analysis.

1. Convert to correct type
2. Consolidate name, region, date

Join same region

region_str <- "africa|asia|canada|latin america (excl mexico)|europe|mexico|middle east|oceania"
inbound_region <- tidy_ntto_inbound_m %>% 
   filter(grepl(region_str, MixRegion)) %>% 
   select(Region=MixRegion, Year, Date, Inbound) %>% 
   group_by(Region, Year, Date) %>% 
   summarise(TotalInbound=sum(Inbound)) %>% 
   ungroup
outbound_region <- tidy_ntto_outbound_m %>% 
   select(Region, Year, Date, Outbound) %>% 
   group_by(Region, Year, Date) %>% 
   summarise(TotalOutbound=sum(Outbound)) %>% 
   ungroup
regional_travel <- inner_join(inbound_region, outbound_region, 
                              by=c("Region"="Region", "Year"="Year", "Date"="Date"))

Challenges

There are several challenges in our project: 1. Due to the problems in the data set such as inconsistency, we have to spend much time in cleaning and re-organizing it, which makes the work tedious and laborious. 2. We need country level data in some graphs; however, what we can acquire from the dataset is region level. In that case, we have to project the data onto the whole region, which makes the analysis not comprehensive and detailed. 3. Shiny is a great tool for creating interactive data visualizations in R; however, we do not have much experience in it, and therefore have to spend time learning it, which is not easy in such a short time.

Analysis

We first visualized the trend of inbound for all region, from the plot we can tell that: 1. There’s a growing trend for most regions. 2. Canada has the highest number 3. Inbound is very seasonal, for example, the peak of Canada always happens on July.

p1 <- inbound_region %>% 
   spread(Region, TotalInbound) %>% 
   filter(Date>'2008-11') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~africa, name='africa', mode='lines', line = list(color="gray", width = 1)) %>%
   add_trace(y=~asia, name='asia', mode='lines', line = list(color="red", width = 1)) %>%
   add_trace(y=~canada, name='canada', mode='lines', line = list(color="orange", width = 1)) %>%
   add_trace(y=~europe, name='europe', mode='lines', line = list(color="pink", width = 1)) %>%
   add_trace(y=~`latin america excl mexico`, name='latin america excl mexico', mode='lines', line = list(color="green", width = 1)) %>%
   add_trace(y=~mexico, name='mexico', mode='lines', line = list(color="purple", width = 1)) %>%
   add_trace(y=~`middle east`, name='middle east', mode='lines', line = list(color="black", width = 1)) %>%
   add_trace(y=~oceania, name='oceania', mode='lines', line = list(color="blue", width = 1)) 
p2 <- outbound_region %>% spread(Region, TotalOutbound) %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~africa, name='africa', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~asia, name='asia', mode='lines', line = list(color="red", width = 1), showlegend=F) %>%
   add_trace(y=~canada, name='canada', mode='lines', line = list(color="orange", width = 1), showlegend=F) %>%
   add_trace(y=~europe, name='europe', mode='lines', line = list(color="pink", width = 1), showlegend=F) %>%
   add_trace(y=~`latin america excl mexico`, name='latin america excl mexico', mode='lines', line = list(color="green", width = 1), showlegend=F) %>%
   add_trace(y=~mexico, name='mexico', mode='lines', line = list(color="purple", width = 1), showlegend=F) %>%
   add_trace(y=~`middle east`, name='middle east', mode='lines', line = list(color="black", width = 1), showlegend=F) %>%
   add_trace(y=~oceania, name='oceania', mode='lines', line = list(color="blue", width = 1), showlegend=F)
subplot(p1, p2, nrows=2, shareX=T) %>%
  layout(title = "Inbound v.s. Outbound",
         yaxis = list(title = "Inbound"),
         yaxis2 = list(title = "Outbound"),
         legend = list(orientation = 'h')
         )

Then we move to the Outbound of all regions, from graph belowe we observe several things: 1. Mexico is the number one Outbound country. 2. A huge pump happend on 2009 for Mexico. 3. Outbound number of mexico is increasing, however for other regions it seems stable. 4. Like Inbound, Outbound shows a seasonal pattern as well.

Since Canada and Mexico dominate the number of people and the interest of different behaviour per region, we start looking at inbound and outbound per region.

p1 <- regional_travel %>% 
   filter(Region=='africa') %>% 
   plot_ly(x = ~as.POSIXct(Date), height = 1000) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1)) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1)) %>% 
   layout(autosize=F)
p2 <- regional_travel %>% 
   filter(Region=='asia') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p3 <- regional_travel %>% 
   filter(Region=='canada') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p4 <- regional_travel %>% 
   filter(Region=='europe') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p5 <- regional_travel %>% 
   filter(Region=='latin america excl mexico') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p6 <- regional_travel %>% 
   filter(Region=='mexico') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p7 <- regional_travel %>% 
   filter(Region=='middle east') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
p8 <- regional_travel %>% 
   filter(Region=='oceania') %>% 
   plot_ly(x = ~as.POSIXct(Date)) %>%
   add_trace(y=~TotalInbound, type="scatter", name='Inbound', mode='lines', line = list(color="gray", width = 1), showlegend=F) %>%
   add_trace(y=~TotalOutbound, type="scatter", name='Outbound', mode='lines', line = list(color="blue", width = 1), showlegend=F)
subplot(p1, p2, p3, p4, p5, p6, p7, p8, nrows=8) %>%
  layout(title = "Regional Inbound and Outbound",
         yaxis = list(title = "Africa"),
         yaxis2 = list(title = "Asia"), 
         yaxis3 = list(title = "Canada"),
         yaxis4 = list(title = "Europe"),
         yaxis5 = list(title = "Latin America"),
         yaxis6 = list(title = "Mexico"),
         yaxis7 = list(title = "Middle East"),
         yaxis8 = list(title = "Oceania"),
         legend = list(orientation = 'h', x = 0, y = 1.005)
         )

yearly_spend <- tidy_ntto_spend_y %>% 
   filter(Region!='european union', Region!='south-central america', Region!='overseas') %>% 
   mutate(Region=recode(Region, "asia-pacific"="asia"), Spend=Spend*1000000) %>% 
   select(-Missing) %>% 
   arrange(Region, Year, Type, Category) 

yearly_spend %>%
   group_by(Type, Year) %>%
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Type, TotalSpend) %>%
   ungroup %>% 
   plot_ly(x = ~Year) %>%
   add_trace(y=~`Payments (imports)`, type="scatter", name='Payments (imports)', mode = 'lines+markers', line = list(color="blue", width = 1)) %>%
   add_trace(y=~`Receipts (exports)`, type="scatter", name='Receipts (exports)', mode = 'lines+markers', line = list(width = 1)) %>% 
   layout(title = "Yearly Spending", 
          xaxis = list(title = "Year"), yaxis = list(title = "Spend"), 
          legend = list(orientation = 'h', x = 0.5, y = 1.005))

yearly_spend %>%
   group_by(Type, Year) %>%
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Type, TotalSpend) %>%
   plot_ly(x = ~Year, y = ~`Payments (imports)`, type = 'bar', name = 'Payments', marker = list(color = 'rgb(55, 83, 109)')) %>%
   add_trace(y = ~`Receipts (exports)`, name = 'Receipts', marker = list(color = 'rgb(26, 118, 255)')) %>%
   layout(title = 'Yearly Spending',
         xaxis = list(
           title = "",
           tickfont = list(
             size = 14,
             color = 'rgb(107, 107, 107)')),
         yaxis = list(
           title = 'Spend (Billion $)',
           titlefont = list(
             size = 16,
             color = 'rgb(107, 107, 107)'),
           tickfont = list(
             size = 14,
             color = 'rgb(107, 107, 107)')),
         # legend = list(orientation = 'h', x = 0.5, y = 1.005)
         legend = list(orientation = 'h', x = 0, y = 1, bgcolor = 'rgba(255, 255, 255, 0)', bordercolor = 'rgba(255, 255, 255, 0)'),
         barmode = 'group', bargap = 0.15, bargroupgap = 0.1)

'layout' objects don't have these attributes: 'bargroupgap'
Valid attributes include:
'font', 'title', 'titlefont', 'autosize', 'width', 'height', 'margin', 'paper_bgcolor', 'plot_bgcolor', 'separators', 'hidesources', 'smith', 'showlegend', 'dragmode', 'hovermode', 'xaxis', 'yaxis', 'scene', 'geo', 'legend', 'annotations', 'shapes', 'images', 'updatemenus', 'ternary', 'mapbox', 'radialaxis', 'angularaxis', 'direction', 'orientation', 'barmode', 'bargap', 'mapType'
'layout' objects don't have these attributes: 'bargroupgap'
Valid attributes include:
'font', 'title', 'titlefont', 'autosize', 'width', 'height', 'margin', 'paper_bgcolor', 'plot_bgcolor', 'separators', 'hidesources', 'smith', 'showlegend', 'dragmode', 'hovermode', 'xaxis', 'yaxis', 'scene', 'geo', 'legend', 'annotations', 'shapes', 'images', 'updatemenus', 'ternary', 'mapbox', 'radialaxis', 'angularaxis', 'direction', 'orientation', 'barmode', 'bargap', 'mapType'

yearly_spend %>% 
   filter(Type=="Payments (imports)") %>% 
   select(-Type) %>% 
   group_by(Region, Year) %>% 
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Region, TotalSpend) %>% 
   plot_ly(x = ~Year) %>%
   add_trace(y=~africa, type="scatter", name='africa', mode = 'lines+markers', line = list(width = 2)) %>%
   add_trace(y=~asia, type="scatter", name='asia', mode = 'lines+markers', line = list(width = 2)) %>%
   add_trace(y=~europe, type="scatter", name='europe', mode = 'lines+markers', line = list(width = 2)) %>%
   add_trace(y=~`latin america`, type="scatter", name='latin america', mode = 'lines+markers', line = list(width = 2)) %>%
   add_trace(y=~`middle east`, type="scatter", name='middle east', mode = 'lines+markers', line = list(width = 2)) %>% 
   layout(title = "Payments (imports)", xaxis = list(title = "Year"), yaxis = list(title = "Spend"),
          legend = list(orientation = 'h'))

yearly_spend %>% 
   filter(Type=="Receipts (exports)") %>% 
   select(-Type) %>% 
   group_by(Region, Year) %>% 
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Region, TotalSpend) %>% 
   plot_ly(x = ~Year) %>%
   add_trace(y=~africa, type="scatter", name='africa', mode = 'lines+markers', line = list(width = 1)) %>%
   add_trace(y=~asia, type="scatter", name='asia', mode = 'lines+markers', line = list(width = 1)) %>%
   add_trace(y=~europe, type="scatter", name='europe', mode = 'lines+markers', line = list(width = 1)) %>%
   add_trace(y=~`latin america`, type="scatter", name='latin america', mode = 'lines+markers', line = list(width = 1)) %>%
   add_trace(y=~`middle east`, type="scatter", name='middle east', mode = 'lines+markers', line = list(width = 1)) %>% 
   layout(title = "Receipts (exports)", 
          xaxis = list(title = "Year"), 
          yaxis = list(title = "Spend"), 
          legend = list(orientation = 'h'))

yearly_spend %>%
   group_by(Type, Year) %>%
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Type, TotalSpend) %>%
   plot_ly(x = ~Year) %>%
   add_trace(y=~`Payments (imports)`, type="scatter", name='Payments (imports)', mode = 'lines+markers', line = list(color="blue", width = 1)) %>%
   add_trace(y=~`Receipts (exports)`, type="scatter", name='Receipts (exports)', mode = 'lines+markers', line = list(width = 1)) %>% 
   layout(title = "Africa", 
          xaxis = list(title = "Year"), yaxis = list(title = "Spend"), 
          legend = list(orientation = 'h', x = 0.5, y = 1.005))

tidy_wb_gdp %>% 
   filter(CountryCode=="USA", Year>2000, Year<2016) %>% 
   select(Year, GDP) %>% 
   mutate(Year=factor(Year)) %>% 
   plot_ly(x = ~Year) %>%
   add_trace(y=~GDP, type="scatter", name='US', mode = 'lines+markers', line = list(width = 1))

yearly_spend %>% 
   filter(Type=="Receipts (exports)") %>% 
   select(-Type, -Region) %>% 
   group_by(Category, Year) %>% 
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Category, TotalSpend) %>% 
   plot_ly(x = ~Year, y = ~Education, type = 'bar', name = 'Education') %>%
   add_trace(y = ~`Medical/Short-Term Workers`, name = 'Medical/Short-Term Workers') %>% 
   add_trace(y = ~`Other Business/Other Personal Travel`, name = 'Other Business/Other Personal Travel') %>% 
   layout(yaxis = list(title = 'Spend'), barmode = 'group')

yearly_spend %>% 
   filter(Type=="Receipts (exports)") %>% 
   select(-Type, -Region) %>% 
   group_by(Category, Year) %>% 
   summarise(TotalSpend=sum(Spend)) %>% 
   spread(Category, TotalSpend) %>% 
   plot_ly(x = ~Year, y = ~Education, type = 'bar', name = 'Education') %>%
   add_trace(y = ~`Medical/Short-Term Workers`, name = 'Medical/Short-Term Workers') %>% 
   add_trace(y = ~`Other Business/Other Personal Travel`, name = 'Other Business/Other Personal Travel') %>% 
   layout(yaxis = list(title = 'Spend'), barmode = 'group')

Finally, we select several region and combine inbound, outbound with GDP

Conclusion

Discuss limitations and future directions, lessons learned.

A note on style: You are encouraged to be as intellectually honest as possible. That means pointing out flaws in your work, detailing obstacles, disagreements, decision points, etc. – the kinds of “behind-the-scene” things that are important but often left out of reports. You may use the first person (“I”/“We”) or specific team members’ names, as relevant.

The goal of this project is to answer the four questions raised above. We have accomplished the task despite of some flaws and limitations. For instance, the latest data we could find is for the year 2015, which is more than one year ago. In that case, we could not say the analysis is very persuasive since the data source is not up to date. We are unable to get data that are more detailed so that we cannot dig deeper to find root causes of some observations. At the very beginning, we thought the data was exhaustive and comprehensive and we made plenty of hypotheses and expectations based on the first impression; however, we find that most of our ideas are not realistic as we investigate the data more. Unfortunately, we have to discard some interesting ideas due the inconstancy in the dataset. For the future work, we could introduce more supportive data such as the inbound and outbound traveler data of other countries, and then compare it with the USA. We believe that it would reveal results that are more interesting.